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ABSTRACT 



The Monte Carlo method was used, and the factors 
cons idered were (1) level of main effects in the population; (2) 
level of interaction effects in the population; (3) alpha level used 
in determining whether to pool; and (4) number of degrees of freedom- 
The results indicated that when the ratio degrees of freedom 
(axb) /degrees of freedcm (within) was large (1/4) r pooling resulted 
in a disturbance in the actual alpha for the main effect test- The 
magnitude and nature of file disturbance was dependent on the alpha 
level employed in testing the interaction effects. The use of an 
alpha of -25 for the interaction effects resulted in a congruence 
between actual alpha and nominal alpha, and a slight increase in 
power . (Author/CK) 
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Abstract 



The Monte Carlo method was used, and the factors 

considered were (1) level of main effects in the population; 

(2) level of interaction effects in the population; (3) alpha 

level used in determining whether to pool; and (4) number 

of d . f . . The results indicated that when the ratio 

wi cn in 

d.f, axb /d.f * within was lar S e ( 1M) > pooling resulted in 
a disturbance in the actual alpha for che main effect test. 
The magnitude and nature of the disturbance was dependent 
upon the alpha level employed in testing the interaction 
effects. The use of an alpha of ,25 for the interaction 
effects resulted in a congruence between actual alpha and 
nominal alpha, and a slight increase in power. 
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John T. Pohlmann 
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Objectives of the Inquiry 

The purpose of this study was to determine the effects 
of pooling the interaction component with the within 
component, on the actual alpha and power of the resulting 
main effects tests. In this study, only the two-way fixed 
effects ANOVA, with two levels of each main effect, was 
considered. The initial structural mode 1 including 
interaction, was 



X ijk - V + a ± + Pj + + e.^ (1) 

where i = 1 , 2 ; j = 1 , 2 ; k = 1 , * * * ( 2 , 3 , or 4); and 

e ijk as the error term. The revised structural model, 
excluding interaction, was 



3, 



( 2 ) 



±jk i j ijk 

where i = 1, 2; j = 1, 2; k = 1, # * * (2, 3, or A), as in 

model (1), but where e. is the pooled interaction and 

x Jk 

within terms. In terms of sample values, the pooled error 
term is given by 



MS error = ^ SS axb + SS within * 1 (d ' f 'axb + d ' f 'within * 
Decisions about what terms should be included in the 
model are based upon either theory or experience (previous 
research) . In the absence of theory and prior knowledge, a 
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researcher must rely on the data available in his experiment 
as a basis for revising an initial model. Tests undertaken 
to revise a structural model, given the data, are called 
preliminary tests. For example, an insignificant preliminary 
test of the interaction term in model (1), at some specified 
alpha* level, would result in pooling the interaction term 
with the within term to give model (2). 

Previous research in the area of preliminary tests 
suggested that two schools of thought prevail. The first 
school of thought may be characterized as the "never pool" 
school. Proponents of this line of reasoning include 
Ostle (1963), and Scheffe (1959). Ostle (1963) reasoned 
that since pooling results in a disturbance in the actual 
alpha level for other tests in the experiment, the inter- 
action component should never be pooled; no matter what 
the result of the F-test for interaction. Sheffe (1959) 
stated that since little is known aboni- rhe opera' ing 
characteristics of such procedures, researchers should 
try to avoid pooling by designing the experiment such that 
dhere will be a sufficient number of error degrees of 
freedom. 

The other school of thought may be characterized 
as the "sometimes pool" school. Proponents of this position* 
(Bennett & Franklin (1954); Bosovich, Bancroft & Hartley 
(1956); and Singh (1970)) reason that pooling is an acceptable 
procedure as lorag as the researcher is aware of the con- 
sequences of the pooling decision and can specify the 
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conditions under which pooling leads to tests where actual 
alpha and nominal alpha are in congruence, and which increase 
in power. Each of the proponents of the "sometimes pool ,, 
approach offer decision rules which a researcher may use 
in determining whether or not: to pool. Bennett and 
Franklin (1954) state that a researcher should pool the 
interaction SS into error, if and only if the F-test for 
the interaction is not significant at the 5 percent level 
and the F-test is less than 2. Bosovich, Bancroft and 
Hartley (1956), in a study of the mixed effects model 
ANOVA, found that if the F-test for interaction is not 
significant at the .25 level, pooling results in a final 
F-test, when made at a nominal 5 percent level, which is 
very close to the 5 percent level. Singh (1970) found 
that the use of a preliminary alpha of .50 resulted in an 
err term that was least biased. 

This study examined the effects of using various alpha 
levels in the preliminary test on the magnitude of the actual 
alpha and power of the final test of the main effects. 

Further, this study considered the effects of using these 
various preliminary alpha's when model (1) was correct 
(E(MS axb ) > E(MS within >, and when model (2) was correct 

< E « s »*b > - E < Ms „ith ln >- 

Method 

The design for this study was a 4X4X3X6 factorial design. 
The factors wtfere (1) four levels of the magnitude of main 
effects in the population (tf* =* 0, 1, 2, 3); (2) four levels 
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of the magnitude of the interaction effects in the population 
(?$ * 0, 1, 2, 3) ; ( 3 ) three levels of degrees of freedom 
within (4, 8, 12), and (4) six alpha levels for the preliminary 
tests (.00 (always pool), ,05, .10, ,25, .50, and 1.00 
(never pool)). In all cases there were two levels of each 
main effect and consequently d#f. ^ was always 1. Only a 
nominal alpha of ,05 was used in testing the main effects. 

Data Source 

The Monte Carlo method was used in the conduct of this 
investigation. A computer program, written in FORTRAN IV, was 
developed by the author for the purpose of this study. 

Input for the program consisted of (1) the number of levels 



in factors A and B; (2) the number of replications (cell n) ; 
(3) a matrix of treatment effects for each treatment com- 
bination. These treatment effects were added to the random 
numbers as they were drawn from the random number generator. 
IBM subroutine GAUSS was used to generate normally distributed 
random numbers. Subroutine GAUSS sums 12 uniformly random 
values generated by the power residue method in subroutine 
RANDU. GAUSS then performs the appropriate linear trans- 
formations to give a normally distributed random number 
from a population with the parameters required by the user. 

The cell into which the random number was placed, determined 
which treatment effect was administered. The magnitude 
of the treatments effects was determined by the population 
values of the main effects and intereaction effects. Each 
combination of the population values of the main effects ^ 
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interaction effects and cell size was run for 2000 trials. 

The output from the program consisted of (1) the number 
of times, in 2000 trials, thet ^ull hypothesis for the 

main effects was rejected at the .05 level, for each of the 
alpha levels used in the preliminary test. This output 
was converted into a proportion that approximated a limiting 
relative frequency that could be interpreted as a probability 
of rejecting the null hypothesis. When the null hypothesis 
was true with respect to the main effects, this value was 
interpreted as an actual alpha level, and when the null 
hypothesis was false with respect to the main effects, this 
value was interpreted as the power of the test. 

Results 

In Table 1 the results are presented in terms of the 
proportion of times in 2000 trials that the null hypothesis, 
for main effects, was rejected. The nominal alpha was 
.05 for all analyses. 

The results are summarized below according to the 

relative magnitude of d.f. ^ to ^’within ’ anc * t * ie a ^P^ a 
level employed in the preliminary test. These results 
refer to the actual alpha and power associated with the main 
effects test under the various preliminary test conditions. 

1. The Relative Magnitude of d.f.^ to 

A. The largest disturbance in the magnitude of 

actual alpha and power occurred when the ratio 

d.f. / d.f. . was 1/4. 

axb within 
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B. The least disturbance in the magnitude of actual 
alpha and power occurred when the ratio 



II. The Alpha Level Employed in the Preliminary Test 
A. The largest disturbance in actual alpha and 



power occurred under the alpha - .00 (always 
pool) condition. The use of this alpha level 
resulted in a consistent reduction in actual 
alpha and a loss of power, when interaction 
was present in the population. 



1/12, the use of alpha - .25 resulted in a 
congruence between actual alpha and nominal 
alpha, and a slight increase in power when 
the population interaction effect was small 
< 1.0) . When the population interaction 
effect was large (tf > 2.0) actual power and 
nominal power converged. 



Conclusions and Implications 

On the basis of these results. It was concluded that 
(1) when the ratio d.f. axb / <i.f. wlthin is relatively large 
(about 1/4), pooling becomes a questionable procedure due to 
the disturbance in Type I error rate; (2) when the ratio 
d * f • axb / ‘within is relative ^y small (about 1 / 12 ) pooling, 
using a preliminary alpha of .25 results in a congruence 
between actual alpha and nominal alpha and a slight increase 
in the power of the main effects test. These results are 



d.f. i / d . f . « . i « 

axb within 



was 1/12 



B. When the ratio d.f. b 



/ d.f. , .. . was 
within 
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in keeping with those of Bosovich, Bancroft and Hartley 
(1956), in that a preliminary alpha of .25 resulted in main 
effects test that were superior to those when other pre- 
liminary alpha levels were used. 

It is therefore suggested that a researcher, confronted 
with the decision of whether to pool or not, use the flow 
chart presented in Figure 1 to assist him in his decision 
making. 
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Figure 1 

Flow Chart Outlining Pooling Decision Rules 
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Footnote 

* Hs a non-centrality parameter for the Frdistribution . 
In the central F distribution, = 0. As the expected value 
of the F distribution increases, increases proportionally. 
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